{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q-learning解决冰冻湖问题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **记住**: Q-learning是一种无模型，离线策略算法，它使用Q函数去寻找一个优化的动作。Q函数是包含每一对状态和动作的价值的表。\n",
    "    \n",
    "观看Q-learning教程 `英文`|`bilibili`[Q learning explained by Siraj](https://www.bilibili.com/video/av16869146?from=search&seid=15203024211966898887)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Q-learning算法有以下五部分组成:\n",
    "<br>\n",
    "<br>\n",
    "![](img/short_diag.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 问题的现实环境是什么？\n",
    "\n",
    "#### 我们使用Q-leaerning运行在[Gym](http://gym.openai.com/)环境中的一个叫做[冰冻湖游戏](https://gym.openai.com/envs/FrozenLake-v0/)\n",
    "\n",
    "![](img/frozenlake_v0.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 开始编码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting gym\n  Downloading https://files.pythonhosted.org/packages/9b/50/ed4a03d2be47ffd043be2ee514f329ce45d98a30fe2d1b9c61dea5a9d861/gym-0.10.5.tar.gz (1.5MB)\nRequirement already satisfied: numpy>=1.10.4 in c:\\python27\\lib\\site-packages (from gym)\nRequirement already satisfied: requests>=2.0 in c:\\python27\\lib\\site-packages (from gym)\nRequirement already satisfied: six in c:\\python27\\lib\\site-packages (from gym)\nCollecting pyglet>=1.2.0 (from gym)\n  Downloading https://files.pythonhosted.org/packages/1c/fc/dad5eaaab68f0c21e2f906a94ddb98175662cc5a654eee404d59554ce0fa/pyglet-1.3.2-py2.py3-none-any.whl (1.0MB)\nCollecting future (from pyglet>=1.2.0->gym)\n  Downloading https://files.pythonhosted.org/packages/00/2b/8d082ddfed935f3608cc61140df6dcbf0edea1bc3ab52fb6c29ae3e81e85/future-0.16.0.tar.gz (824kB)\nInstalling collected packages: future, pyglet, gym\n  Running setup.py install for future: started\n    Running setup.py install for future: finished with status 'done'\n  Running setup.py install for gym: started\n    Running setup.py install for gym: finished with status 'done'\nSuccessfully installed future-0.16.0 gym-0.10.5 pyglet-1.3.2\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  Cache entry deserialization failed, entry ignored\n  Cache entry deserialization failed, entry ignored\n  Cache entry deserialization failed, entry ignored\n  Cache entry deserialization failed, entry ignored\n  Cache entry deserialization failed, entry ignored\n  Cache entry deserialization failed, entry ignored\n  Cache entry deserialization failed, entry ignored\n  Cache entry deserialization failed, entry ignored\n  Cache entry deserialization failed, entry ignored\nYou are using pip version 9.0.1, however version 18.0 is available.\nYou should consider upgrading via the 'python -m pip install --upgrade pip' command.\n"
     ]
    }
   ],
   "source": [
    "## 安装gym环境\n",
    "!pip install gym"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "import random\n",
    "from collections import namedtuple\n",
    "import collections\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 根据不同的策略选择动作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def select_eps_greedy_action(table, obs, n_actions):\n",
    "    '''\n",
    "    Select the action using a ε-greedy policy (add a randomness ε for the choice of the action)\n",
    "    '''\n",
    "    value, action = best_action_value(table, obs)\n",
    "\n",
    "    if random.random() < epsilon:\n",
    "        return random.randint(0,n_actions-1)\n",
    "    else:\n",
    "        return action\n",
    "\n",
    "def select_greedy_action(table, obs, n_actions):\n",
    "    '''\n",
    "    Select the action using a greedy policy (take the best action according to the policy)\n",
    "    '''\n",
    "    value, action = best_action_value(table, obs)\n",
    "    return action\n",
    "\n",
    "\n",
    "def best_action_value(table, state):\n",
    "    '''\n",
    "    Exploring the table, take the best action that maximize Q(s,a)\n",
    "    '''\n",
    "    best_action = 0\n",
    "    max_value = 0\n",
    "    for action in range(n_actions):\n",
    "        if table[(state, action)] > max_value:\n",
    "            best_action = action\n",
    "            max_value = table[(state, action)]\n",
    "\n",
    "    return max_value, best_action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/Q_function.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Q_learning(table, obs0, obs1, reward, action):\n",
    "    '''\n",
    "    Q-learning. Update Q(obs0,action) according to Q(obs1,*) and the reward just obtained\n",
    "    '''\n",
    "    \n",
    "    # Take the best value reachable from the state obs1\n",
    "    best_value, _ = best_action_value(table, obs1)\n",
    "\n",
    "    # Calculate Q-target value \n",
    "    Q_target = reward + GAMMA * best_value\n",
    "\n",
    "    # Calculate the Q-error between the target and the previous value\n",
    "    Q_error = Q_target - table[(obs0, action)]\n",
    "\n",
    "    # Update Q(obs0,action)\n",
    "    table[(obs0, action)] += LEARNING_RATE * Q_error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 测试策略"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_game(env, table, n_actions):\n",
    "    '''\n",
    "    Test the new table playing TEST_EPISODES games\n",
    "    '''\n",
    "    reward_games = []\n",
    "    for _ in range(TEST_EPISODES):\n",
    "        obs = env.reset()\n",
    "        rewards = 0\n",
    "        while True:\n",
    "            # Act greedly \n",
    "            next_obs, reward, done, _ = env.step(select_greedy_action(table, obs, n_actions))\n",
    "            obs = next_obs\n",
    "            rewards += reward\n",
    "\n",
    "            if done:\n",
    "                reward_games.append(rewards)\n",
    "                break\n",
    "\n",
    "    return np.mean(reward_games)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 主程序"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Some hyperparameters..\n",
    "GAMMA = 0.95\n",
    "\n",
    "# NB: the decay rate allow to regulate the Exploration - Exploitation trade-off\n",
    "#     start with a EPSILON of 1 and decay until reach 0\n",
    "epsilon = 1.0\n",
    "EPS_DECAY_RATE = 0.9993\n",
    "\n",
    "LEARNING_RATE = 0.8\n",
    "\n",
    "# .. and constants\n",
    "TEST_EPISODES = 100\n",
    "MAX_GAMES = 15001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tEp: 0 Test reward: 0.0 1.0\n",
      "\tEp: 1000 Test reward: 0.0 0.5\n",
      "\tEp: 2000 Test reward: 0.0 0.25\n",
      "\tEp: 3000 Test reward: 0.0 0.12\n",
      "\tEp: 4000 Test reward: 0.31 0.06\n",
      "\tEp: 5000 Test reward: 0.59 0.03\n",
      "\tEp: 6000 Test reward: 0.64 0.01\n",
      "\tEp: 7000 Test reward: 0.71 0.01\n",
      "\tEp: 8000 Test reward: 0.72 0.0\n",
      "\tEp: 9000 Test reward: 0.47 0.0\n",
      "\tEp: 10000 Test reward: 0.74 0.0\n",
      "\tEp: 11000 Test reward: 0.75 0.0\n",
      "\tEp: 12000 Test reward: 0.7 0.0\n",
      "\tEp: 13000 Test reward: 0.81 0.0\n",
      "\tEp: 14000 Test reward: 0.79 0.0\n",
      "\tEp: 15000 Test reward: 0.72 0.0\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABJIAAAJQCAYAAADVHU6aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd8lfXd//H3lb0TQhIyyGSFvRLCCOBAVNS6KrIcLNu7t9rWtr9q7667u71/tbW/jrsM0YqCiyquOtBKAmQhsoeQnRCSEMjOSc451+8PkCJSDZKT65zk9Xw88pBzcnmddx4acq53vt/PZZimKQAAAAAAAODzeFkdAAAAAAAAAJ6BIgkAAAAAAADdQpEEAAAAAACAbqFIAgAAAAAAQLdQJAEAAAAAAKBbKJIAAAAAAADQLRRJAAAAAAAA6BaKJAAAAAAAAHQLRRIAAAAAAAC6xcfqAJcqKirKTElJsToGAAAAAABAn7Fz58560zSjP+84jyuSUlJSVFRUZHUMAAAAAACAPsMwjLLuHMfWNgAAAAAAAHQLRRIAAAAAAAC6hSIJAAAAAAAA3UKRBAAAAAAAgG6hSAIAAAAAAEC3UCQBAAAAAACgWyiSAAAAAAAA0C0USQAAAAAAAOgWiiQAAAAAAAB0C0USAAAAAAAAuoUiCQAAAAAAAN1CkQQAAAAAAIBuoUgCAAAAAABAt1AkAQAAAAAAoFsokgAAAAAAANAtFEkAAAAAAADoFookAAAAAAAAdAtFEgAAAAAAALqFIgkAAAAAAADdQpEEAAAAAACAbqFIAgAAAAAAQLdQJAEAAAAAAKBbfKwOAAAAAACAOzJNU4++fUTbj53U8EGhGhkXqpFxYRoRG6qwAF+r4wGWoEgCAAAAAOACTqepH27ep/V55RoVF6bX9x7XhoLyc59PiAjUyLhQpceGKf1swZQyMFjeXoaFqQHXo0gCAAAAAOA8Tqep77+8T8/kl+urs4fou9eNkCSdaLLpYE2TDh5v0qHjzTpU06T3DtfJ4TQlSf4+XhoRG6r02PMKptgwDQj2s/LLAXoURRIAAAAAAGc5nab+66W92lBQoa9dMUTfuXaEDOPMKqPY8ADFhgfoyhEx54632R06WtuiQ8ebzxRMNc3acrBWzxVVnjsmNixA6WdXL328iiktOli+3owthudxaZFkGMZ1kh6T5C1pjWmav7rg80mSnpQUcfaYh03TfN2VmQAAAAAAuBin09Qjm/bq2aIK3X/lUH1r7vBzJdK/4+/jrdHx4RodH/6J5+uabTp03uqlgzXN2na0WF2OM6uX/Ly9NDQm5NyqpY+3x0WF+Lvs6wN6gmGapmtObBjeko5IukZSpaRCSQtN0zxw3jGrJO0yTfMvhmGMkvS6aZopn3XejIwMs6ioyCWZAQAAAAD9k9Np6rsv7tHzOyv14FVD9c1rPr9EulRdDqeO1Z1dvVTzr+1xJ5ps546JCvE/u2rpX9vjhsaEyN/Hu0ezABcyDGOnaZoZn3ecK1ckTZF01DTN4rOBNkq6WdKB844xJYWd/XO4pGoX5gEAAAAA4FMcZ0ukF3ZW6utXD9M3rxnuktfx9fY6Uw7FhukWJZx7vqG1U4eON+lgTbMOnd0e9+SOMnXanZIkHy9DQ6JDzm2P+3gV06Aw/x4vu4DP48oiKUFSxXmPKyVlXXDMjyW9ZRjGA5KCJc252IkMw7hP0n2SlJSU1ONBAQAAAAD9k8Np6jsv7NamD6r0jTnD9I05rimRPktksJ+mD43S9KFR556zO5wqPdmqg2dXLR063qyi0lN6+cN/rb8YEOT7iWIpPS5UwweFKsCX1UtwHauHbS+U9IRpmr81DGOapKcMwxhjmqbz/INM01wlaZV0ZmubBTkBAAAAAH2Mw2nq28/v1t93Vemha4brwauHWR3pHB9vLw2NCdXQmFDdND7+3PONbV1niqWa5rMzmJq1saBC7V0OSZKXIaVGBSs9Lkwjz9selxARyOol9AhXFklVkhLPezz47HPnWy7pOkkyTXOHYRgBkqIk1bowFwAAAACgn7M7nPrW87v18ofV+vbc4br/KvcpkT5LeJCvstIGKitt4LnnHE5T5Q1tn9get7eyUa/tOX7umNAAn3Orlj4ul0YMClWwv9XrS+BpXPl/TKGkYYZhpOpMgbRA0qILjimXdLWkJwzDGCkpQFKdCzMBAAAAAPo5u8Oph57brc27q/Wda0foP68canWky+LtZSg1KlipUcG6fmzcueebO7p05ETzJ7bHbfqgSi22MkmSYUjJkUHniqX02DCNjAtV4oAgeXmxegkX57IiyTRNu2EY90t6U5K3pMdN09xvGMZPJBWZprlZ0rckrTYM45s6M3j7XtNVt5EDAAAAAPR7dodT33j2Q72657i+e126/uOKIVZHcpnQAF9NTo7U5OTIc885naaqTrfr4PFPbo9780CNPr4aD/bz1ojY0H9tj4sL04jYUIUF+Fr0lcCdGJ7W22RkZJhFRUVWxwAAAAAAeJguh1Pf2PihXtt7XI9cn66vzO67JdKlauu068iJlnN3jTtwvEmHjjepqcN+7piEiECNjAvTkqlJumJEjIVp4QqGYew0TTPj845jMyQAAAAAoM/rcjj19Y279PreGv3XvJFaOSvN6khuJcjPRxMSIzQhMeLcc6Zp6nhjx7lVS4dqmrWztEH3rivUl8bH6wc3jlJ0qL+FqWEFiiQAAAAAQJ/W5XDqgWd26R/7a/T9G0ZqxUxKpO4wDEPxEYGKjwjUVemDJEk2u0N/+ecx/fm9Y3r/SJ2+Ny9d8zMSuSNcP+JldQAAAAAAAFyl0+7U/c98oH/sr9EPbxxFiXSZ/H289Y05w/X617M1YlCovvviXi1YladjdS1WR0MvoUgCAAAAAPRJnXan/vOZD/Tm/hP68U2jtCw71epIfcbQmFBtvG+qfnXbWB083qTrf5+jx975SDa7w+pocDGKJAAAAABAn2OzO/S1p3fq7QMn9JObR+veGZRIPc3Ly9CCKUl651uzNXf0IP3unSO64Q+5KixtsDoaXIgiCQAAAADQp9jsDn1t/Qd652CtfnrzaN09LcXqSH1aTGiA/rhoktbdm6n2Tofu+N8demTTXjW2d1kdDS5AkQQAAAAA6DM6uhz66lM7teVQrX52yxjdRYnUa65Mj9HbD83SypmperawXHMefV+v7qmWaZpWR0MPokgCAAAAAPQJHV0OfXX9Tr13uE6/uHWslkxNtjpSvxPk56P/umGUNt+frUFh/rr/mV1a/mSRKk+1WR0NPYQiCQAAAADg8Tq6HLrvqZ365+E6/eq2sVqUlWR1pH5tTEK4XvraDH3/hpHaceyk5v5uq9bmlsjhZHWSp6NIAgAAAAB4tI4uh1b+rUg5H9XpN7eP04IplEjuwMfbSytmpumtb85SVmqkfvrqAd3yp23aV9VodTRcBookAAAAAIDHau90aMWTRco9Wq9f3z5O8zMTrY6ECyRGBunxezP1x0UTdbyxQzf/aZt+/toBtXXarY6GL4AiCQAAAADgkdo7HVrxt0JtO1av//nyeM3PoERyV4Zh6MZx8dry0GzNzxis1TkluubRrXrvcK3V0XCJKJIAAAAAAB6nrdOuZU8Uavuxk/rtHeP15cmDrY6EbggP8tUvbxun574yTQG+Xlq6rlAPbNilumab1dHQTRRJAAAAAACP8nGJlF9yUo/OH6/bJlEieZopqZF6/esz9Y05w/Tmvhpd/dt/6tnCcpkmw7jdHUUSAAAAAMBjtNrsunddoQpKGvS7Oyfo1omUSJ7K38db35gzXK9/fabS48L03Rf36s5VeTpa22J1NHwGiiQAAAAAgEdotdm1dF2hikob9PsFE3XzhASrI6EHDI0J0caVU/Xr28fq0PEmzXssR4+985FsdofV0XARFEkAAAAAALfXYrPr3nUF2ll+So8tmKgvjY+3OhJ6kJeXoTszk7TlW1foujGx+t07RzTvsRwVlDRYHQ0XoEgCAAAAALi15o4u3fN4gT4oP60/LJiomyiR+qzoUH/9YeFErVuaqY4up+b/dYce2bRHjW1dVkfDWRRJAAAAAAC39XGJtLvitP64cKJuGBdndST0gitHxOjth2bpvllperawQlc/+r5e2V3NMG43QJEEAAAAAHBLTR1duvvxAu2pbNQfF03U9WMpkfqTID8ffW/eSG2+P1tx4QF6YMMuLXuiUJWn2qyO1q9RJAEAAAAA3E5TR5fuXlugvZWN+tPiSbpuDCVSfzUmIVx//9p0/eDGUcovadA1j27Vmpxi2R1Oq6P1SxRJAAAAAAC30tjepbvWFmh/daP+vHiSrh0da3UkWMzH20vLs1P11jdnadqQgfrZawd1y5+3aV9Vo9XR+h2KJAAAAACA22hs69Jda/N1oLpRf1k8WXMpkXCewQOCtPaeDP1x0UTVNNr0pT/m6uevHVBbp93qaP0GRRIAAACAfqujy6GKhjbtqTytFhsXolZrbOvSkrX5OnS8Wf+7ZLLmjBpkdSS4IcMwdOO4eG15aLbuzEzS6pwSXfPoVr13qNbqaP2C4WkTzzMyMsyioiKrYwAAAABwU+2dDtW32FTbbFN9y9mP5s5//bnFpvqWTtU329R8XnkU6u+jhVlJund6iuIjAi38Cvqn022dWrI2X0dqWvTXuybryvQYqyPBQxSWNuiRTXt1tLZFN42P1w9vHKXoUH+rY3kcwzB2mqaZ8bnHUSQBAAAAcHetNrvqW2yqO1sO1Z0tgj5RDLXYVN9sU2un46LniAjyVVSIv6JC/M7+01/RoWcehwb46o19NXp973EZkm4cF6cVM9M0JiG8d7/QfupUa6cWr8nX0boWrbprsq4YQYmES2OzO/TX94v1x3ePKsDXS9+bN1LzMxLl5WVYHc1jUCQBAAAAcFumaarFZj9XAH1cENU3ny2JWj65mqi96+LlUGSw3yeKoagQf0WFnnkcfd7jgcH+8vP5/MkeFQ1tWretVM8Wlqu106HpQwZq5cw0zR4ezQWpizScLZGO1bVo9d0Zmj082upI8GDH6lr0vU17lV/SoCmpkfrFrWM1NCbE6lgegSIJAAAAQK8yTVNNHfbzCiHb2VVD/yqGzl9JZLN/+tbdhiENDD6/GDr759BPPo4J9VdksJ98vF0z9rWxvUsbC8q1blupapo6NCwmRCtmpurmCQkK8PV2yWv2Rw2tnVq0Ok8l9a1afXeGZlEioQeYpqnniyr189cPqr3Toa9dOUT/ccUQ+fvwvftZKJIAAAAAXDbTNNXY3nV21VDneeXQp7eU1bd0qtPx6XLI28s4u3LoTBEUfa4YunCL2ZlyyNuNVv502p16bW+1Vm0t0cHjTYoK8dc905K1ZGqyBgT7WR3Po51ssWnxmnyV1Ldq7T2Zyh4WZXUk9DF1zTb99NUD2ry7WkOig/XL28ZpSmqk1bHcFkUSAAAAgH/L4TR1rK7l3MqhugtWDn28pexkq01djk9fM/h4GRp4kS1l0eeVQh8XRwOC/Dx+W5hpmtp+7KRWbS3W+0fqFODrpTsmJ2p5dqpSooKtjudx6ltsWrw6X2UNZ0qkGUMpkeA6/zxcq++/tE+Vp9q1cEqiHr5upMKDfK2O5XYokgAAAAD8Wz98eZ/+tqPsE8/5ehsX3VJ2/gqij+cOhQf6enw59EUdrmnWmpxivfxhtbqcTs0dNUj3zUrT5GRWOnRHXbNNi1bnqeJUmx6/N1PTh1AiwfXaOu36/TsfaW1uiQYE+elHN43SjePiZBj98++xi6FIAgAAAHBRTR1dyvr5FmUPi9Ly7NRzg6nDAn24qLoEtc0d+tv2Mj2VV6bG9i5NTIrQyplpunZ0rFttz3Mntc0dWrQ6X1Wn2vX4vZmaNmSg1ZHQz+yratQjm/Zqb1WjrhwRrZ/eMkaDBwRZHcstUCQBAAAAuKgnt5fqR5v3a/P9MzRucITVcTxeW6ddzxdVam1uicob2pQUGaRlM1J0R0aigv19rI7nNmqbOrRwdZ6ON3Zo3b2ZykqjRII1HE5TT24v1f9967BMU/rW3OG6d3qKy4b3ewqKJAAAAACfYpqm5v5uqwL9vLX5/myr4/QpDqeptw/UaNXWYn1Qflrhgb5anJWke6enKCYswOp4lqpt6tCC1XmqaezQE0unMPAYbqHqdLt++NI+bTlUqzEJYfrVbeM0JiHc6liWoUgCAAAA8Cn5xSd156o8/eb2cZqfmWh1nD5rZ1mDVm8t0ZsHauTjZejmCQlaOTNNI2JDrY7W6040dWjhqjydaOrQE8umKDOFEgnuwzRNvbGvRj/avF8nW2xaNiNV37xmeL9cTUiRBAAAAOBTHtiwS+8frlX+9+Yo0M/b6jh9Xml9qx7fVqLniyrV3uXQrOHRum9mmmYMHdgv5lHVNJ7ZzlbXbNOTyzIZSA631djepd/845Cezi9XQkSgfnbLGF2ZHmN1rF5FkQQAAADgE+qabZr+qy26a2qKfnjTKKvj9Cun2zr1dH651m0rVX2LTemxoVo5M003jY+Xn0/fnMtyvLFdC1flqb6lU08um6LJyQOsjgR8rsLSBj2yaa+O1rboxnFx+uFNoxQT2j+2plIkAQAAAPiEP713VP/z5mFt+dZsDYkOsTpOv2SzO/Tyh9VavbVYH9W2aFCYv+6dnqpFWUkKD/S1Ol6PqT7droWr89TQ0qknl0/RpCRKJHgOm92hv75frD++e1QBvl763ryRmp+RKK8+fjdGiiQAAAAA5zicpmb95j0lDwzSMyunWh2n3zNNU+8fqdPqnGJtO3pSwX7emp+ZqGUzUpUY6dm3Iq86fWYl0qm2Tj21PEsTErkzIDzTsboWfW/TXuWXNGhKSqR+cdsYDY3pu3POKJIAAAAAnLPl4Aktf7JIf148SfPGxlkdB+fZX92oNTklemV3tZymqevHxmnlzDSPLGAqT7Vp4eo8nW7r0vrlWRrvgV8DcD7TNPV8UaV+/vpBtXc69B9XDNHXrhwif5++N2OOIgkAAADAOUvXFWh/dZO2PXyVfL375kweT3e8sV1PbC/VM3nlarbZNSUlUitnpenq9BiP2FJT0XCmRGpq79L6FVkaN5gSCX1HfYtNP331gF7+sFpDooP1i1vHKittoNWxehRFEgAAAABJZy7wZ/3Pe3rgyqF6aO4Iq+Pgc7TY7Hq2sEKP55ao6nS70qKCtSw7VV+ePFgBvu65CqKioU0LVuWpxWbX0yuyNCYh3OpIgEu8f6RO339pryoa2rUgM1GPXD9S4UF9Y74ZRRIAAEAf5nCaOtlqU31zp+pbbOd9dKq+2aa6s38+2WLTkqnJevDqYVZHhoV+/Y9D+uv7x5T73asUHxFodRx0k93h1Bv7arQ6p1h7KhsVGeynJVOTdfe0ZEWF+Fsd75zyk2dWIrV22rV+OSUS+r72Tod+v+WI1uSUaECQn579ytQ+cQMDiiQAAAAPY3c4dbK1U3XN55VCLTbVX/C4rtmmhrZOXextnJ+Pl6JD/BUV6q/oED9Vne5Q2clW7Xjk6j51Ryh0n83u0PRfvqtJyQO0+u7PvT6AGzJNUwUlDVqdU6x3DtbKz8dLt09K0PLsNA2NsfbitexkqxauylNbl0NPr8jS6HhKJPQf+6sb9dSOMv381rHy9oDtp5+nu0WST2+EAQAA6K867c5PrByq+3jl0EVWEjW0dl70HAG+XooO9VdUiL8SI4M0MWmAokP8FHX2uTMfZx6H+vvIMP71ZnZ/daNu+EOuNhaU6yuzh/TWlw038o99NTrZ2qklU5OtjoIvyDAMZaUNVFbaQB2tbdHa3BJt+qBSGwoqdHV6jFbOSlNWauQnvvd7Q2l9qxauzlNHl0PPrJiqUfFhvfr6gNVGx4frV7ePszpGr2NFEgAAwCWy2R3ntpCdXwTVXfC4vsWm021dFz1HsJ/3eUWQ378KobMric5/HOznfVkXiAtX5an0ZKu2/p8rGbLcD83/3x060dyh9751hUcMbEb3nGyx6am8Mv1tR5kaWjs1NiFcK2amat7YuF75Pi+pP7MSqdPh1NMrsjQyjhIJ8HRsbQMAALgEHV2Oz9xS9q+VRDY1ddgveo5Qf5+z5dB5RVCIv6JC//U4+uzjIL/eWxj+8W3f/7Bwor40Pr7XXhfWO1zTrGt/v1Xfm5eu+2axIq0v6uhyaNMHVVqTU6zi+lYlRARq6YwU3ZmZqNAA12xnLa5r0cLVeepymHpmZZbSYymRgL6ArW0AAKDfa+u0q775vBLo32wpq2+2qdl28XIoLMDn3MqhkbFhihrqd26l0PmriaJD/d32bkpXjohRWlSw1uYU66Zxcb2+/QXWWZ9XJj8fL90xOdHqKHCRAF9vLcpK0oLMRL17qFarcor1s9cO6rF3PtLCrCTdOz2lRwesH6tr0cJVeXI4TW1YOVUjYkN77NwAPANFEgAA6DNsdof++n6xNn1Qqbpmm1o7HRc9LiLI91wJNDo+7FwRdOEWs4HBfm5bDl0KLy9DS7NT9YOX9mln2SllpERaHQm9oNVm1993VenGsXEaEOxndRy4mJeXoTmjBmnOqEHaXXFaq3OKtTa3RI/nlujGcXFaMTPtsu+mdrT2zEok0zS14b6pGj6IEgnojyiSAABAn1BQ0qBHNu3RsbpWzRoeravSB53bUhb98bayUH9FBvvJz6f/zQm6fVKCfvvWYa3JKaFI6ide+rBKLTa7FjNku98ZnxihPy6apIqGNq3bVqpnC8v10ofVmj5koFbOStMVw6MveWXi0dpmLViVL0nasHKqhlEiAf0WRRIAAPBojW1d+tU/DmpDQYUGDwjUE0szdcWIGKtjuZ0gPx8tmpKk/33/mMpPtilpYJDVkeBCpmlqfV65RsaFaVJShNVxYJHEyCD98KZR+vqcYdpYUK5120q1dF2hhsWEaMXMVN08IaFbqy4/OtGshavzZBiGNqycqqExIb2QHoC76n+/jgMAAH2CaZp6ZXe1rn70fT1XVKn7ZqXprW/OokT6DPdMT5G3l6F120usjgIX+6D8tA4eb9KSqUnMxILCA331ldlDtPX/XKnf3TlePt5e+u6Le5X96/f0/7Z8pFOtnf/23z1c06wFq/LkZRjaeB8lEgAXF0mGYVxnGMZhwzCOGobx8EU+/zvDMD48+3HEMIzTrswDAAD6hspTbVr2RKEe2LBLceEBevk/Z+h780b26p3QPNGgsADdOC5ezxVWqKmjy+o4cKGn88oU4u+jWyYkWB0FbsTPx0u3Thys1x/M1tMrsjQ6Pky/ffuIpv1qi37w0j6V1rd+4vhDNU1atDpPPt5nSqQh0ZRIAFy4tc0wDG9Jf5J0jaRKSYWGYWw2TfPAx8eYpvnN845/QNJEV+UBAACez+5w6ontpfrtW0dkGNIPbhyle6Yly8ebRdbdtTw7VX/fVaVnCyq0claa1XHgAqdaO/Xq3uO6MyNRwf6Uq/g0wzA0Y2iUZgyN0uGaZq3JKdazhRVan1+muaMG6b5ZaQry89HiNfny8/bShvumKjUq2OrYANyEK3+yTJF01DTNYkkyDGOjpJslHfg3xy+U9CMX5gEAAB5sX1WjHt60R/uqmnRVeox+essYJfTgLa37izEJ4cpKjdQT20u1dEYKJVwf9PzOCnXanVrCkG10w4jYUP3PHeP1netG6G/by/RUXpne3H9Cvt6GokL8tWHlVKVQIgE4jyvfOSRIqjjvceXZ5z7FMIxkSamS3v03n7/PMIwiwzCK6urqejwoAABwX22ddv38tQP60h9zdaLJpj8tmqS192RQIl2GFTPTVHW6Xf/YX2N1FPQwp9PU0/nlmpISqRGx3FUL3RcTGqBvXztCOx65Sj+5ebSuGBGjjfdRIgH4NHdZ67pA0gumaTou9knTNFdJWiVJGRkZZm8GAwAA1nnvUK2+/9I+VZ1u16KsJH33unSFB/paHcvjXZ0eo5SBQVqbW6Ibx8VbHQc9KOdovcpOtumha4ZbHQUeKsjPR3dPS9Hd01KsjgLATblyRVKVpMTzHg8++9zFLJC0wYVZAACAB6lrtumBDbu09IlCBfp56/mvTtMvbh1LidRDvLwMLctO1a7y09pZdsrqOOhB6/PKNDDYT9eNibU6CgCgj3JlkVQoaZhhGKmGYfjpTFm0+cKDDMNIlzRA0g4XZgEAAB7A6TS1saBcV//2n3pzX40euma4XnswW5kpkVZH63NunzRYYQE+ejy3xOoo6CHVp9u15eAJzc9MlL+Pt9VxAAB9lMu2tpmmaTcM435Jb0rylvS4aZr7DcP4iaQi0zQ/LpUWSNpomiZb1gAA6MeO1rboe3/fq4KSBmWlRuoXt43lVtMuFOzvo0VZyVq19ZgqGtqUGBlkdSRcpo0F5TIlLZqSZHUUAEAf5tIZSaZpvi7p9Que++EFj3/sygwAAMC92ewO/eWfx/Tn944p0M9bv7l9nO7IGCzDMKyO1ufdMz1Za3KK9eT2Un3/xlFWx8Fl6HI4tbGwQlcMj6YUBAC4lLsM2wYAAP1QQUmDHtm0R8fqWnXzhHj94MZRigrxtzpWvxEXHqgbxsVpY2GFvj5nmEIDmEHlqd4+cEK1zTb9cmqy1VEAAH2cK2ckAQAAXFRjW5ce2bRH8/+6Qza7U08szdRjCyZSIllgeXaqWmx2PVdUaXUUXIb1eWVKiAjUFSNirI4CAOjjWJEEAAB6jWmaenXPcf33Kwd0qq1T981K0zfmDFOQH29JrDJucISmpERq3bYS3TMtWT7e/J7R0xyra9H2Yyf1nWtHyNuLLaEAANfinQIAAOgVlafatOyJQj2wYZfiwgP08n/O0PfmjaREcgPLslNVeapdbx04YXUUfAFP55XL19vQnZmJVkcBAPQDvHMDAAAuZXc49cT2Uv32rSMyDOmHN47SPdNTWDnhRq4ZNUhJkUFam1uieWPjrI6DS9De6dALOyt03Zg4toYCAHoFRRIAAHCZfVWNenjTHu2ratLV6TH6yS1jlBARaHUsXMDby9DSGSn671cOaFf5KU1MGmB1JHTTK7ur1dRh15KsJKujAAD6Cba2AQCAHtdqs+tnrx7Ql/6YqxNNNv158SStuSeDEsmN3ZGRqNAAH63NLbGntyiZAAAgAElEQVQ6Ci7B+vwyDR8UoimpkVZHAQD0E6xIAgAAPeq9Q7X6/kv7VHW6XYuykvTd69IVHsht5d1diL+PFk5J0trcElWdbqf08wB7Kk9rT2Wj/vtLo2UYbBUFAPQOViQBAIAeUdvcofuf+UBLnyhUoJ+3nv/qNP3i1rGUSB7knukpkqQnt5damgPdsz6vTIG+3rp1UoLVUQAA/QgrkgAAwGVxOk09V1ShX7x+UB1dTj10zXB9ZXaa/H28rY6GS5QQEajrx8RqQ365Hrx6mEL8eavorhrburR5d7VunZigsADKWgBA72FFEgAA+MKO1jZrwao8Pbxpr0bGhemNb8zUg1cPo0TyYCtmpqnZZtfzRRVWR8FnePGDSnV0ObU4K9nqKACAfoZfMwEAgEtmszv05/eO6S//PKZAP2/95vZxuiNjMHNa+oAJiRGanDxAj28r0d3TUuTtxX9Td2Oapp7OL9PEpAiNSQi3Og4AoJ9hRRIAALgk+cUnNe+xHD225SNdPzZWW741W/MzEymR+pAV2amqaGjX2wdOWB0FF7Gj+KSO1bVqCauRAAAWYEUSAADolsa2Lv3yjYPaWFihwQMC9eSyKZo9PNrqWHCBuaNjlRgZqLW5xbpuTKzVcXCB9Xlligjy1Q3j4qyOAgDohyiSAADAZzJNU6/sOa6fvHJAp9o69ZVZafr6nGEK8uNtRF/l7WXo3ump+umrB7S74rTGJ0ZYHQln1TZ16K39J7R0RooCfJlFBgDofWxtAwAA/1ZFQ5uWPlGoBzfsUnxEgDbfP0OPzBtJidQPzM8YrFB/H63NLbE6Cs6zsbBCdqepRWxrAwBYhCIJAAB8it3h1JqcYs393VYVlDTohzeO0t+/NkOj4xns21+EBvjqzsxEvb73uKpPt1sdBzrzfbmhoFwzh0UpNSrY6jgAgH6KIgkAAHzC3spG3fLnbfrZawc1fchAvf3QbC3LTuXuXf3QvTNS5DRNPbmj1OookPTuoVodb+zQYlYjAQAsxLp0AAAgSWq12fXo20e0bluJBob468+LJ+n6MbHcja0fGzwgSNePidOG/HI9eNUwBfvz1tFK6/PLFRsWoDkjY6yOAgDox1iRBAAA9O6hE5r7u61am1uihVOS9M5DszVvbBwlErR8ZqqaOux6YWel1VH6tbKTrdp6pE4LpiTKx5u38AAA6/BrJQAA+rHa5g799ysH9Nqe4xoWE6IXvjpNGSmRVseCG5mUNEATkyK0bluJlkxNZoujRZ7JL5e3l6GFU5KsjgIA6Of4dQYAAP2Q02lqQ0G55vz2fb194IS+dc1wvfbgTEokXNSK7DSVnmzTloMnrI7SL3V0OfRcUYXmjhqkQWEBVscBAPRzrEgCAKCfOVrbrEc27VVh6SlNTYvUL24dq7ToEKtjwY1dO3qQEiICtTa3RHNHx1odp995fe9xnWrr0pKpDNkGAFiPIgkAgH7CZnfoz+8d05//eVTB/j76zZfH6Y7Jg5mDhM/l4+2lpTNS9LPXDmpvZaPGDg63OlK/sj6vTGlRwZo+ZKDVUQAAYGsbAAD9QV7xSV3/WI4e2/KRbhgbp3cemq35GYmUSOi2+ZmJCvbz1trcYquj9CsHqpv0QflpLcpK4vsVAOAWKJIAAOjDGtu69PCLe7RgVZ66HE49uWyKfr9goqJC/K2OBg8TFuCrOzOT9Oqe46pp7LA6Tr+xPr9M/j5e+vLkwVZHAQBAEkUSAAB9kmma2ry7Wlc/+k89v7NSX5mdpre+MVuzh0dbHQ0ebOmMFDlNU0/uKLU6Sr/Q3NGll3ZV6abx8YoI8rM6DgAAkpiRBABAn1PR0Kbvv7RP7x+p0/jB4Xpy2RSNjmemDS5fYmSQrh0dq2fyy/XAVUMV5MdbSVd6aVeV2jodDNkGALgVViQBANBH2B1Ord5arLm/26qi0gb96KZR2vS1GZRI6FHLs1PV2N6lF3dWWh2lTzNNU+vzyjU2IVzjGW4OAHAj/BoJAIA+YG9lox7etEf7q5s0Z2SMfnLzGMVHBFodC33Q5OQBGp8Yoce3lWpxVrK8vBgA7QpFZad0+ESzfn37WIZsAwDcCkUSAAAeyjRNFZQ0aHVOid45eEIxof76y+JJum5MLBeecBnDMLQ8O1UPbtildw/Vas6oQVZH6pPW55UpNMBHN42PtzoKAACfQJEEAICHsTucemNfjVbnFGtPZaMig/304NXDtDw7VeGBvlbHQz9w/ZhYxYcHaG1uCUWSC9S32PT63uNanJXMHCoAgNvhJxMAAB6ixWbXs4UVejy3RFWn25UaFayf3zpGt08arABfb6vjoR/x9fbSPdNT9Ms3Dml/dSNzuHrYc0UV6nKYWjI1yeooAAB8CkUSAABu7nhju57YXqpn8svV3GHXlJRI/fhLo3V1egzzaWCZBVOS9NiWj7Q2t0SPzp9gdZw+w+E09Ux+uaamRWpoTKjVcQAA+BSKJAAA3NSB6iatySnW5t3Vcpqm5o2N08qZaRqfGGF1NEDhgb6an5Gop/PL9PB16YoJC7A6Up+w9UidKk+16+Hr062OAgDARVEkAQDgRkzT1PtH6rQmp0S5R+sV5Oetu6elaOmMFCVGBlkdD/iEpTNS9OSOUv1tR5m+fe0Iq+P0CevzyhQV4q+5o2KtjgIAwEVRJAEA4AZsdode/rBaa3KKdeREiwaF+evh69O1cEoSA7ThtpIHBuuakYO0Pr9M/3nlUAX6MavrclSeatO7h2v1n1cMlZ+Pl9VxAAC4KIokAAAsdLqtU0/nl+uJ7aWqa7YpPTZUj84frxvHxXMhCY+wYmaa3jpwQpt2VWpxVrLVcTzahoJyGZIWZjFkGwDgviiSAACwQNnJVj2eW6LniirV3uXQrOHR+t38NM0YOlCGwQBteI7MlAEaNzhca3NLtDAziQHwX1Cn3alnCyt0VfogJUQEWh0HAIB/iyIJAIBetLPslNbkFOsf+2vk42Xo5gkJWjEzVemxYVZHA74QwzC0PDtVX9/4od4/Uqcr02OsjuSR3txfo/qWTi2ZymokAIB7o0gCAMDFHE5Tbx+o0eqcEu0sO6XwQF/9x+whumd6igZxpyv0AfPGxumXrx/SmtxiiqQv6Km8MiVFBmnWsGirowAA8JkokgAAcJG2Trte2FmptbklKjvZpsTIQP34plG6IyNRwf78CEbf4evtpXump+jX/zikg8ebNDKOFXaX4siJZhWUNOjh69PZGggAcHu8iwUAoIfVNnfob9vLtD6/TKfbujQxKULfvS5d146OlTcXieijFk1J0h+2fKS1uSX6v3eMtzqOR3k6r0x+3l66Y/Jgq6MAAPC5KJIAAOghR040a01OsV7aVa0up1NzRw3SfbPSNDk50upogMuFB/nqjozB2lhQof9z3QjFhLJtsztabXZt+qBK88bGamCIv9VxAAD4XBRJAABcBtM0tf3YSa3OKdY/D9cpwNdLd2Ymanl2qlKigq2OB/SqpTNS9VRemdbvKNNDc0dYHccjbN5drWabXUumJlsdBQCAbqFIAgDgC+hyOPXqnmqt3lqiA8ebFBXir29dM1xLpiZrQLCf1fEAS6RGBevq9EFan1+ur105VAG+3lZHcmumaWp9XpnSY0M1OXmA1XEAAOgWiiQAAC5BU0eXNuSX64ntpTre2KGhMSH69e1jdfOEBC6aAUkrZqZqwaoT+vuuKi2cwq3sP8uHFae1v7pJP7tljAyD+WkAAM9AkQQAQDdUnmrTum2l2lhQrtZOh6YPGahf3DpWs4dHc5cl4DxZqZEaHR+mtbklWpCZSEHyGdbnlSvYz1u3TEywOgoAAN1GkQQAwGfYU3laq3NK9Pre45Kkm8bFacXMNI1JCLc4GeCeDMPQipmp+uazu/X+kTpdMSLG6khu6VRrp17dU607MgYrxJ+35AAAz8FPLQAALuB0mnr3UK1W5xQrv6RBof4+Wp6dqnunpyg+ItDqeIDbu2FsvH75+iGtzS2hSPo3XthZKZvdyZBtAIDHoUgCAOCsji6HNn1QpTW5xSqua1VCRKC+f8NI3ZmZqNAAX6vjAR7Dz8dL90xP0f+8eViHa5o1IjbU6khuxek09XR+mTKSByg9NszqOAAAXBKKJABAv3eyxaan8sr01I4ynWzt1JiEMD22YILmjY2Tr7eX1fEAj7Q4K0n/792PtDa3WL/58nir47iVbcfqVXqyTd+YM9zqKAAAXDKXFkmGYVwn6TFJ3pLWmKb5q4scM1/SjyWZknabprnIlZkAAPjYsboWrc0t0Ytnt5hcnR6jFTPTNDUtkgHBwGWKCPLTlycP1nNFlfrOtemKDvW3OpLbWJ9XpshgP10/NtbqKAAAXDKXFUmGYXhL+pOkayRVSio0DGOzaZoHzjtmmKRHJM0wTfOUYRhsogcAuJRpmiooadDqnBJtOXRCvt5eun1SgpZnp2poDNtvgJ60dEaq1ueVa31emb55DatvJKmmsUPvHKzVipmp8vfxtjoOAACXzJUrkqZIOmqaZrEkGYaxUdLNkg6cd8xKSX8yTfOUJJmmWevCPACAfszucOqNfTVak1Os3ZWNGhDkqweuGqa7piazUgJwkSHRIbo6PUbr88r0H1cMUYAvxcmGgnI5TVOLpzBkGwDgmVxZJCVIqjjvcaWkrAuOGS5JhmFs05ntbz82TfMfF57IMIz7JN0nSUlJSS4JCwDom1psdj1bWKHHc0tUdbpdqVHB+tktY3T7pMEK9OOiFnC15dmpWrQmXy9/WKU7M/v3+7guh1MbC8s1e3i0kgYGWR0HAIAvxOph2z6Shkm6QtJgSVsNwxhrmubp8w8yTXOVpFWSlJGRYfZ2SACA56lp7NC67SV6Jr9czR12ZaYM0I9uGqU5IwfJy4v5R0BvmTZkoEbGhWltbonmZyT26/ljWw6e0Ikmm35+C6uRAACey5VFUpWkxPMeDz773PkqJeWbptklqcQwjCM6UywVujAXAKAPO1DdpDU5xdq8u1pO09T1Y+O0cmaaJiRGWB0N6JcMw9Dy7FR9+/ndyvmoXrOGR1sdyTLr88qVEBGoK9MZCwoA8FyuLJIKJQ0zDCNVZwqkBZIuvCPbS5IWSlpnGEaUzmx1K3ZhJgBAH2Sapt4/Uqc1OSXKPVqvID9v3TUtWctmpCoxku0jgNVuGh+nX//jkNbmlvTbIqm4rkW5R+v17bnD5c2qSACAB3NZkWSapt0wjPslvakz848eN01zv2EYP5FUZJrm5rOfm2sYxgFJDknfMU3zpKsyAQD6FpvdoZc/rNbanBIdPtGsQWH++u516Vo0JUnhQb5WxwNwlr+Pt+6emqzfvn1EH51o1rBB/e8OiU/nl8vHy9D8zMTPPxgAADdmmKZnjRzKyMgwi4qKrI4BALDY5t3V+umrB1TXbFN6bKhWzkzTTePj5efjZXU0ABfR0Nqpab/cotsmJeiXt42zOk6v6uhyKOsXW5Q9LEp/WjTJ6jgAAFyUYRg7TdPM+LzjrB62DQDAJWvvdOi//r5XiQOC9Oj88coeGtWvB/gCniAy2E+3TRqsFz+o1LfnjtDAEH+rI/WaV3ZXq7G9S0uyGLINAPB8/NoWAOBxXtt7XM0ddv3gxlGaOSyaEgnwEMuzU9Rpd2p9XrnVUXrV+vxyDY0J0dS0SKujAABw2SiSAAAeZ2NBuVKjgrkoAzzM0JhQXTkiWk/llaqjy2F1nF6xr6pRuytOa3FWEqU3AKBPoEgCAHiUj040q6jslBZkJnJRBnig5dlpqm/p1Obd1VZH6RXr88oU6Out2yYNtjoKAAA9giIJAOBRNhRUyNfb0O2TuSgDPNGMoQOVHhuqx3NL5Gk3fblUje1devnDat08IV7hgdxJEgDQN1AkAQA8RkeXQ5t2VWruqFhF9aNBvUBfYhiGlmWn6lBNs7YdPWl1HJfa9EGl2rscWjKVIdsAgL6DIgkA4DHe3F+j021dWjAl0eooAC7DzRPiFRXir7W5xVZHcRnTNPV0frnGJ0ZoTEK41XEAAOgxFEkAAI+xoaBciZGBmjEkyuooAC6Dv4+37pqarPcO1+lobbPVcVwir7hBR2tbtCQryeooAAD0KIokAIBHKKlvVV5xgxZkJsnLiyHbgKdbMjVJfj5eenxbqdVRXGJ9fpnCA3110/h4q6MAANCjKJIAAB5hY2G5vL0M3cGQbaBPGBjir9smJujFnZVqaO20Ok6Pqm3u0Jv7avTlyYMV4OttdRwAAHoURRIAwO112p16oahSV6XHKCYswOo4AHrIsuxU2exOPZNfZnWUHvVcYYXsTlOL2dYGAOiDKJIAAG7vnYMndLK1U4umcFEG9CXDB4Vq1vBoPbmjTDa7w+o4PcLhNLWhoEIzhg5UWnSI1XEAAOhxFEkAALe3oaBc8eEBmjU82uooAHrYiuxU1TXb9Oru41ZH6RHvHapV1el23TU12eooAAC4BEUSAMCtVTS0Keejet2RkShvhmwDfc7MYVEaPihEa3JLZJqm1XEu2/r8Mg0K89eckYOsjgIAgEtQJAEA3NqzhRXyMqT5mYlWRwHgAoZhaHl2qg4eb9KO4pNWx7ks5Sfb9P6ROi3ITJKPN2+zAQB9Ez/hAABuy+5w6rmiCs0eHq2EiECr4wBwkZsnJGhgsJ/W5pRYHeWyPF1QJi/D0ELmuQEA+jCKJACA23r3UK1qm21clAF9XICvt5ZMTdaWQ7UqrmuxOs4XYrM79HxRpeaMjFFsOHeXBAD0XRRJAAC3tbGwQjGh/roqPcbqKABcbMnUZPl5e+nxbZ65KumNvTVqaO3UEoZsAwD6OIokAIBbqj7drn8ertUdGYOZNQL0A9Gh/rplYrxe2Fmp022dVse5ZOvzypQyMEgzhkRZHQUAAJfinTkAwC09V1QhpyktyGRbG9BfLMtOVUeXU0/nl1sd5ZIcqmlSUdkpLc5Klhd3lwQA9HEUSQAAt+NwmnqusEIzh0UpMTLI6jgAekl6bJhmDovS33aUqtPutDpOt63PK5O/j5e+PHmw1VEAAHA5iiQAgNvZeqRO1Y0drEYC+qFl2ak60WTTa3urrY7SLS02u/7+QZVuHBevAcF+VscBAMDlKJIAAG5nQ0G5Bgb76ZpRg6yOAqCXzR4WraExIVqTUyLTNK2O87le2lWl1k6Hlkyl+AYA9A8USQAAt1Lb1KEth2r15cmD5efDjymgv/HyMrRsRqr2Vzcpv6TB6jifyTRNrc8r0+j4ME1IjLA6DgAAvYJ36AAAt/L8zko5nKbuzEy0OgoAi9w2KUEDgny1JqfE6iifaWfZKR2qadaSqckyDIZsAwD6B4okAIDbcDpNbSws19S0SKVFh1gdB4BFAny9tWRqsrYcOqGS+lar4/xb6/PKFOrvo5snxFsdBQCAXkORBABwG9uPnVRFQ7sWTmHWCNDf3TUtWb5eXlq3zT1XJZ1ssen1vTW6bVKCgvx8rI4DAECvoUgCALiNDQXligjy1bWjY62OAsBiMaEBuml8vJ4vqlRjW5fVcT7l+Z2V6nQ4tXhqstVRAADoVRRJAAC3UN9i01sHanTbxMEK8PW2Og4AN7A8O1XtXQ49U1BudZRPcDpNPZNfrimpkRo+KNTqOAAA9CqKJACAW3hxZ6W6HKYWTmHINoAzRsWHacbQgXpye6m6HE6r45yz9aM6lTe06S5WIwEA+iGKJACA5UzT1LOFFcpIHqBh/HYfwHmWZ6eqpqlDr+89bnWUc9bnlSsqxJ9tuACAfokiCQBgufySBhXXt2oBQ7YBXOCK4TFKiw7W2twSmaZpdRxVnW7Xu4dO6M7MwfLz4a00AKD/4acfAMByGwrKFRrgoxvGxlkdBYCb8fIytGxGqvZUNqqw9JTVcbQhv1ymxN0lAQD9FkUSAMBSp9s69ca+Gt06MUGBfgzZBvBpt08arIggX63NLbY0R6fdqY2FFbpqRIwGDwiyNAsAAFahSAIAWGrTB1XqtDu1IJPf7gO4uEA/by3OStJbB06o7GSrZTneOlCj+habljBkGwDQj1EkAQAsY5qmNhSUa/zgcI2KD7M6DgA3dve0FPl4GVq3rdSyDOvzyjR4QKBmDY+2LAMAAFajSAIAWOaD8lP6qLaFWSMAPtegsADdNC5ezxVVqLG9q9df/2hts/KKG7QoK0neXkavvz4AAO6CIgkAYJkNBRUK9vPWTePjrY4CwAMsy05VW6dDzxaW9/prr88rl6+3ofkZib3+2gAAuBOKJACAJRrbu/Tqnmp9aUK8gv19rI4DwAOMSQjX1LRIPbGtVHaHs9det63Trhc/qNS8sXGKCvHvtdcFAMAdUSQBACyx+cMqdXQ52dYG4JKsyE5TdWOH3thX02uv+cruajV32BmyDQCAKJIAABYwTVPPFFRoVFyYxiaEWx0HgAe5Kj1GqVHBWpNTLNM0Xf56pmnqqbwyjRgUqozkAS5/PQAA3B1FEgCg1+2pbNTB401amJUkw2BoLYDu8/IytGxGinZXNmpn2SmXv97uykbtq2rSkqn8fQUAgESRBACwwMbCcgX6euvmCQzZBnDpbp88WOGBvlqbW+Ly11qfV6YgP2/dMjHB5a8FAIAnoEgCAPSqFptdL39YrRvGxSkswNfqOAA8UJCfjxZlJenN/TWqaGhz2eucbuvUK7urdcvEBIXy9xUAAJIokgAAveyV3dVq63QwZBvAZblnWoq8DEPrtpW67DVe2Fkpm92pJVkM2QYA4GMUSQCAXrWxoFzDB4VoUlKE1VEAeLDY8ADdOC5OzxaWq6mjq8fPb5qmnskv16SkCI2KD+vx8wMA4KkokgAAvWZ/daN2VzZqQSZDawFcvuXZaWrtdOi5wooeP/f2YydVXN+qJVNZjQQAwPkokgAAvWZjQYX8fLx02ySG1gK4fGMHh2tKaqTWbSuV3eHs0XOvzyvTgCBfzRsb16PnBQDA01EkAQB6RXunQy/tqtK8MbGKCPKzOg6APmJ5dqqqTrfrzf0neuycJ5o69NaBE5qfkagAX+8eOy8AAH0BRRIAoFe8uqdazTa7FjBkG0APmjNykJIHBmlNbnGPnXNjQYUcTlOLsvj7CgCAC1EkAQB6xcbCCqVFBSsrNdLqKAD6EG8vQ0unp2hX+WntLDt12eezO5zaUFCuWcOjlTwwuAcSAgDQt1AkAQBc7siJZu0sO6UFUxIZsg2gx92RkajQAB89nlty2ed652Ctapo6tITVSAAAXJRLiyTDMK4zDOOwYRhHDcN4+CKfv9cwjDrDMD48+7HClXkAANbYUFAuX29Dt08abHUUAH1QsL+PFk1J0hv7jquioe2yzvV0fpniwgN0VXpMD6UDAKBvcVmRZBiGt6Q/Sbpe0ihJCw3DGHWRQ581TXPC2Y81rsoDALBGR5dDf99VpbmjYzUwxN/qOAD6qHump8gwDD25vfQLn6OkvlU5H9Vr4ZQk+XizcB8AgItx5U/IKZKOmqZZbJpmp6SNkm524esBANzQm/trdLqtSwsz2SYCwHXiIwJ1w9g4PVtYoeaOri90jmfyy+TjZWhBZmIPpwMAoO9wZZGUIKnivMeVZ5+70O2GYewxDOMFwzAu+lPbMIz7DMMoMgyjqK6uzhVZAQAu8kx+uZIigzR9yECrowDo45Znp6rZZtdzRZWX/O92dDn0/M5KzR09SDFhAS5IBwBA32D1mt1XJKWYpjlO0tuSnrzYQaZprjJNM8M0zYzo6OheDQgA+OKK61qUX9KgOzMT5eXFkG0ArjU+MUKZKQP0/9u79ygrz/s+9N+HGe4gIQnEHUmWZFnoBjbCd8f3yLEiUBLbqMmJ3aR1s1acpE3axj3tcnt82q6m6UnOWY1XTt1ce2ILy04Esq1YVnxJYicWyGLQ/YItCZgBBJJAAonrPOcPxu1YBrFBs+fde+bzWWvW7Pfd7+z5sta7ZmZ/eZ/f+8fffjzHButpfe2X792RvS8cyc+94YI2pQOAsaGdRVJ/kuFXGC0a2vc/1VqfrrUeGtr8gySva2MeAEbZ2o3b0jOh5AOvM2QbGB2/+JaLsv3ZF/PVB3ae1tf92V1P5uI50/PGV7l6EgBeTjuLpI1JLi2lXFRKmZRkTZLbhh9QSpk/bPOGJA+1MQ8Ao+jw0cH8+Xe3592Xn2+ZCDBq3rN0XhafOzV/+K3HW/6a+/v3ZdPWvfnZ11+QUlw9CQAvp21FUq31aJKPJbkjxwuiW2qtD5RSPllKuWHosF8tpTxQStmc5FeTfKRdeQAYXXc+uCtPHzicNSsN2QZGT8+Ekn/4poty95PPpm/b3pa+5jN3PZkpEyfkp109CQCn1NYZSbXW22utr661Xlxr/Q9D+z5Ra71t6PG/qrVeUWu9ptb6jlrrw+3MA8DouXnD1iycNTVvu9RsO2B0ffDaxZk5ubelq5KeO3gk6zYN5IZrFuTsqRNHIR0AdLemh20DMAZtffqFfGvLnnxwxeL0GLINjLIZk3uzZuXi3H7fjvTvffFlj731nv68eOSYIdsA0CJFEgAj7nN3b82EknzwWstEgGZ8+E0XJkn+9O+eOOkxtdb82XeezNWLzs7Vi2aNTjAA6HKKJABG1JFjg7nl7u15+2XnZ/7ZU5uOA4xTi86ZluuunJebN2zN/kNHT3jMhsefyWNP7c/Pvd7VSADQKkUSACPq6w8/ld3PH8pNhmwDDftHb7kozx88ms/fve2Ez//ZXVtz1pTe/OQ1C0Y5GQB0L0USACNq7YatmXvW5LzjMkO2gWYtX3JOXrtkVv7420/k2GD9oed2P38oX7l/R37mdYszdVJPQwkBoPsokgAYMf17X8w3H92dD65YnN4ev2KA5v2jt74qW595IXc+uOuH9t9y97YcOVbzs29w9SQAnA5/5QMwYm7ZeHz5yAdXLG44CcBx7106NwtnTc0ffevx/7nv2GDNZ+/amjddfF4unjOjwXQA0H0USQCMiGODNbfcvS1vuWR2FjV/T5YAACAASURBVJ87rek4AEmS3p4J+YdvvjAbnngm927fmyT55iNPpX/vi/m5NxiyDQCnS5EEwIj460efyo59B/MPDNkGOsyHrl2cGZN784dDVyX92XeezPkzJ+c9S+c2nAwAuo8iCYARcfOGbZk9Y1Ledbk3ZkBnmTllYj507eJ8+d4d2fjEM/nmo7uz5trFmWiWGwCcNr89AXjFdj13MF9/+Kn89OsWZVKvXy1A5/nImy7MYK356P+4OyXJGldPAsAZ8dc+AK/Y5+/elmODNWuu9cYM6EyLz52W666cl2dfOJJ3XT43C2ZNbToSAHQlRRIAr8jgYM3ajdvyxledl4tmT286DsBJ/ZO3XZxJvRPyi2+5qOkoANC1epsOAEB3+9aWPdn+7Iv5Fz9+WdNRAF7WNYtn5b5/995M7u1pOgoAdC1XJAHwiqzduDXnTJuYH79iXtNRAE5JiQQAr4wiCYAztvv5Q/nqA7vyU69dlCkTvTkDAICxTpEEwBn783u25+hgzU0rFzcdBQAAGAWKJADOSK01n9u4LddeeE4uOX9m03EAAIBRoEgC4Ix85/vP5PE9B7Lm2iVNRwEAAEaJIgmAM3Lzhq05a0pv3n/1/KajAAAAo0SRBMBpe/bA4Xzl/p25cflCQ7YBAGAcUSQBcNr+/J7tOXxsMGtWWtYGAADjiSIJgNNSa83ajduybPGsXD7/rKbjAAAAo0iRBMBp+e6Tz2bLU/tz08rFTUcBAABGmSIJgNPy2Q1bM31ST66/ekHTUQAAgFGmSAKgZftePJLb79uRVcsXZvrk3qbjAAAAo0yRBEDL1vf15+CRwdx0rSHbAAAwHimSAGhJrTWfvWtrrlhwVq5adHbTcQAAgAYokgBoyebt+/Lwzudz00pXIwEAwHilSAKgJWs3bM3UiT1ZtcyQbQAAGK9OWSSVUv6vUsoVoxEGgM60/9DR3LZ5ID95zfzMnDKx6TgAAEBDWrki6aEkny6l3FVK+aVSisEYAOPMbX0DeeHwsayxrA0AAMa1UxZJtdY/qLW+OcnPJ7kwyb2llM+WUt7R7nAAdIa1G7fmsrkzs3zxrKajAAAADWppRlIppSfJa4Y+9iTZnOTXSylr25gNgA5wf/++3Lt9X25auTillKbjAAAADeo91QGllN9Ncn2Sryf5j7XWDUNP/VYp5ZF2hgOgeWs3bs3k3gm5cfmipqMAAAANO2WRlOTeJP+m1nrgBM+tHOE8AHSQFw4fzbpNA/mJq+bn7GmGbAMAwHjXytK2vRlWOJVSZpVSVidJrXVfu4IB0Lwv3bsj+w8dzU2GbAMAAGmtSPq3wwujWuveJP+2fZEA6BRrN2zNxXOm59oLz2k6CgAA0AFaKZJOdEwrS+IA6GKP7Hw+92zdmzXXLjFkGwAASNJakXR3KeV3SikXD338TpLvtjsYAM26ecPWTOqZkJ9+nSHbAADAca0USb+S5HCSzw19HEryy+0MBUCzDh45lls39ee9V8zNudMnNR0HAADoEKdcojZ0t7aPj0IWADrEX96/I/tePGLINgAA8ENOWSSVUuYk+ZdJrkgy5Qf7a63vbGMuABp084ZtueC8aXnjq85rOgoAANBBWlna9pkkDye5KMn/keSJJBvbmAmABn1v9/5sePyZfOjaxZkwwZBtAADgf2mlSDqv1vqHSY7UWv+61voLSd7Q5lwANGTthq3pnVDyM4ZsAwAAL3HKpW1Jjgx93lFKeX+SgSTeXQCMQYeOHsuf39Ofd18+N+fPnHLqLwAAAMaVVoqkf19KOTvJbyT5r0nOSvLP2poKgEbc+eCuPHPgcNasXNx0FAAAoAO9bJFUSulJcmmt9UtJ9iV5x6ikAqARN2/YmoWzpuatl85pOgoAANCBXnZGUq31WJIbRikLAA168ukD+faWp/Ohaxenx5BtAADgBFpZ2vZ3pZTfS/K5JAd+sLPWek/bUgEw6tZu3JYJJfnACmPwAACAE2ulSHrT0OdPDttXk7xz5OMA0IQjxwbz+bu3552vOT/zz57adBwAAKBDnWpG0oQkv19rvWWU8gDQgK899FT27D+UNdcuaToKAADQwU41I2kwycfO9MVLKdeVUh4ppWwppXz8ZY776VJKLaWsONPvBcCZu3nD1sw9a3Lefpkh2wAAwMm9bJE05M5Syj8vpSwupZz7g49TfdHQHd8+leR9SZYmuamUsvQEx81M8mtJ7jrN7ACMgO3PvpC/eWx3PrRicXp7Wvm1AAAAjFetzEj6haHPvzxsX03yqlN83cokW2qt30+SUsraJKuSPPiS4/7PJL+V5F+0kAWAEXbL3duTJB+8dnHDSQAAgE53yiKp1nrRGb72wiTbhm1vT/L64QeUUl6bZHGt9cullJMWSaWUjyb5aJIsWWJ+B8BIOXpsMLds3Ja3XToni86Z1nQcAACgw52ySCql/PyJ9tda/8cr+cZDg7x/J8lHTnVsrfXTST6dJCtWrKiv5PsC8L/89aO7s/O5g/l3N/zIymMAAIAf0crStmuHPZ6S5F1J7klyqiKpP8nwdRKLhvb9wMwkVyb5ZiklSeYlua2UckOt9e4WcgHwCt28YVtmz5icd10+t+koAABAF2hladuvDN8upcxK8qctvPbGJJeWUi7K8QJpTZJ/MOx19yWZPex1v5nknyuRAEbHzn0H8/WHd+Wf/NjFmWjINgAA0IIzeedwIMmrT3VQrfVoko8luSPJQ0luqbU+UEr5ZCnlhjP4vgCMoM/fvS2DNVljyDYAANCiVmYkfTHH79KWHC+elia5pZUXr7XenuT2l+z7xEmOfXsrrwnAKzc4WLN247a86eLzcsF505uOAwAAdIlWZiT9l2GPjyZ5sta6vU15ABgFf7tlT/r3vpiPv+81TUcBAAC6SCtF0tYkO2qtB5OklDK1lHJhrfWJtiYDoG3Wbtiac6ZNzHuvMGQbAABoXSszkj6fZHDY9rGhfQB0od3PH8qdD+7KT792USb39jQdBwAA6CKtFEm9tdbDP9gYejypfZEAaKcvfHd7jg7WrFm5pOkoAABAl2mlSNo9/C5rpZRVSfa0LxIA7TI4WPO5jVuz8sJzc8n5M5qOAwAAdJlWZiT9UpLPlFJ+b2h7e5Kfb18kANrlO99/Ok88/UJ+9V2XNh0FAADoQqcskmqt30vyhlLKjKHt/W1PBUBb3LxxW86a0pufuGp+01EAAIAudMqlbaWU/1hKmVVr3V9r3V9KOaeU8u9HIxwAI+eZA4dzx/0781OvXZQpEw3ZBgAATl8rM5LeV2vd+4ONWuuzSX6ifZEAaIe/uGd7Dh8bzJqVi5uOAgAAdKlWiqSeUsrkH2yUUqYmmfwyxwPQYWqtuXnD1ixfMiuvmXdW03EAAIAu1cqw7c8k+Vop5Y+TlCQfSfKn7QwFwMi6+8ln873dB/Kff/rqpqMAAABdrJVh279VStmc5N1JapI7klzQ7mAAjJyb79qaGZN7c/01hmwDAABnrpWlbUmyK8dLpA8keWeSh9qWCIARte+FI/nyfTuyatmCTJvUyoWoAAAAJ3bSdxSllFcnuWnoY0+SzyUptdZ3jFI2AEbAur7+HDo6mJtWLmk6CgAA0OVe7r+mH07yt0mur7VuSZJSyj8blVQAjIgfDNm+auHZuXLh2U3HAQAAutzLLW37qSQ7knyjlPLfSynvyvFh2wB0ib5te/PwzuezZuXipqMAAABjwEmLpFrrulrrmiSvSfKNJP80yfmllN8vpbx3tAICcOZu3rA1Uyf25IZrFjQdBQAAGANOOWy71nqg1vrZWutPJlmUZFOS32x7MgBekecPHskXN+/IDdcsyMwpE5uOAwAAjAGt3rUtSVJrfbbW+ula67vaFQiAkXHb5oG8eOSYZW0AAMCIOa0iCYDucfOGrXnNvJlZtnhW01EAAIAxQpEEMAbd378v9/c/l5tWLkkp7pMAAACMDEUSwBh084atmdw7IauXLWw6CgAAMIYokgDGmAOHjmZ930Def/X8nD3NkG0AAGDkKJIAxpgv37sj+w8dzU0rlzQdBQAAGGMUSQBjzM0bt+aS82dkxQXnNB0FAAAYYxRJAGPIwzufy6ate7Pm2sWGbAMAACNOkQQwhqzdsC2Teibkp167qOkoAADAGKRIAhgjDh45lr+4Z3t+/Mp5OXf6pKbjAAAAY5AiCWCMuP2+HXnu4NHctHJx01EAAIAxSpEEMEas3bAtF543LW981XlNRwEAAMYoRRLAGLDlqeez4Yln8qFrlxiyDQAAtI0iCWAMWLthW3onlPzM6wzZBgAA2keRBNDlDh09lj+/Z3ves3Ru5syc3HQcAABgDFMkAXS5Ox7YlWdfOJKbVi5pOgoAADDGKZIAutzaDVuz6Jypecsls5uOAgAAjHGKJIAu9sSeA/m77z2dD61YnAkTDNkGAADaS5EE0MXWbtyWngklH1ixuOkoAADAOKBIAuhSR44N5gvf3Z53XHZ+5p09pek4AADAOKBIAuhSX3toV/bsP5SbVroaCQAAGB2KJIAu9dkN2zL/7Cn5sVfPaToKAAAwTiiSALrQtmdeyN8+tjsfWLE4vT1+lAMAAKPDuw+ALnTL3duSJB9csajhJAAAwHiiSALoMkePDeaWu7flx149J4vOmdZ0HAAAYBxRJAF0mW8+sju7njuUNdcuaToKAAAwziiSALrMzRu2ZvaMyXnX5ec3HQUAABhnFEkAXWTHvhfzjUeeygdXLMpEQ7YBAIBR5l0IQBf5/N3bM1iTD127uOkoAADAOKRIAugSxwZrPrdxW95yyexccN70puMAAADjkCIJoEt8a8ue9O99MWtWuhoJAABohiIJoEv8xT3bc/bUiXnP0rlNRwEAAMYpRRJAFzhw6Gi++sCuvP/q+Znc29N0HAAAYJxSJAF0ga8+uDMvHjmWG5cvbDoKAAAwjrW1SCqlXFdKeaSUsqWU8vETPP9LpZT7Sil9pZRvlVKWtjMPQLe6ddNAFs6amtctOafpKAAAwDjWtiKplNKT5FNJ3pdkaZKbTlAUfbbWelWtdVmS/5zkd9qVB6BbPfX8wXzrsd1ZvXxBJkwoTccBAADGsXZekbQyyZZa6/drrYeTrE2yavgBtdbnhm1OT1LbmAegK31p844M1mT1MsvaAACAZvW28bUXJtk2bHt7kte/9KBSyi8n+fUkk5K880QvVEr5aJKPJsmSJUtGPChAJ1vX158rFpyVS+fObDoKAAAwzjU+bLvW+qla68VJfjPJvznJMZ+uta6ota6YM2fO6AYEaND3du/Pvdv3GbINAAB0hHYWSf1JFg/bXjS072TWJlndxjwAXWf9pv6UkvzkNQuajgIAANDWImljkktLKReVUiYlWZPktuEHlFIuHbb5/iSPtTEPQFeptWZd30DefPHszD1rStNxAAAA2jcjqdZ6tJTysSR3JOlJ8ke11gdKKZ9Mcnet9bYkHyulvDvJkSTPJvlwu/IAdJt7tu7N1mdeyK+885KmowAAACRp77Dt1FpvT3L7S/Z9YtjjX2vn9wfoZuv7+jO5d0Kuu3Je01EAAACSdMCwbQB+1JFjg/ni5oG8e+nczJwysek4AAAASRRJAB3pbx7dnWdfOJIbl7lbGwAA0DkUSQAdaF3fQGZNm5i3vXpO01EAAAD+J0USQIfZf+ho7nxwZ66/en4m9foxDQAAdA7vUAA6zB3378zBI4NZbVkbAADQYRRJAB1mXV9/Fp0zNa+74JymowAAAPwQRRJAB3nquYP59pY9uXH5wpRSmo4DAADwQxRJAB3kts0DGazJKsvaAACADqRIAugg6/sGctXCs3PJ+TOajgIAAPAjFEkAHWLLU/tzX/++rFq2oOkoAAAAJ6RIAugQ6/v6M6EkN1yjSAIAADqTIgmgA9Rac+um/rz5ktk5/6wpTccBAAA4IUUSQAf47pPPZvuzL2a1IdsAAEAHUyQBdIB1ff2ZMnFCfvzKeU1HAQAAOClFEkDDDh8dzJfu3ZH3LJ2XGZN7m44DAABwUookgIb9zaO7s/eFI7lxuSHbAABAZ1MkATTs1r7+nDt9Ut566ZymowAAALwsRRJAg54/eCR/9eCuXH/1/Ezs8SMZAADobN61ADToK/fvzKGjg1nlbm0AAEAXUCQBNGh930CWnDstr10yq+koAAAAp6RIAmjIrucO5tvf25PVyxaklNJ0HAAAgFNSJAE05IubB1Jrsmq5ZW0AAEB3UCQBNOTWTf25etHZuXjOjKajAAAAtESRBNCAx3Y9nwcGnstqQ7YBAIAuokgCaMC6vv5MKMn118xvOgoAAEDLFEkAo2xwsGbdpoG85dI5OX/mlKbjAAAAtEyRBDDKvrv12fTvfTE3Ll/QdBQAAIDTokgCGGW3burP1Ik9ee/SeU1HAQAAOC2KJIBRdPjoYL58746894q5mT65t+k4AAAAp0WRBDCKvvnIU9n34hF3awMAALqSIglgFK3vG8h50yflLZfObjoKAADAaVMkAYyS5w4eyZ0P7cr1V8/PxB4/fgEAgO7jnQzAKPnK/Ttz+OhgVi+3rA0AAOhOiiSAUbJuU38uOG9ali2e1XQUAACAM6JIAhgFO/cdzN9//+msXrYwpZSm4wAAAJwRRRLAKLhtc39qjWVtAABAV1MkAYyCWzcN5JrFs3LR7OlNRwEAADhjiiSANntk5/N5aMdzuXHZgqajAAAAvCKKJIA2W9fXn54JJddfo0gCAAC6myIJoI0GB2tu6xvIWy+dndkzJjcdBwAA4BVRJAG00cYnnkn/3hezepkh2wAAQPdTJAG00bq+gUyb1JP3XjG36SgAAACvmCIJoE0OHT2WL987kPcunZtpk3qbjgMAAPCKKZIA2uQbD+/OcwePZvVyy9oAAICxQZEE0Cbr+/oze8akvOWS2U1HAQAAGBGKJIA22PfikXztoady/dUL0tvjRy0AADA2eHcD0AZfuX9HDh8bzI2WtQEAAGOIIgmgDW7d1J+LZk/P1YvObjoKAADAiFEkAYywgb0v5q7Hn8nqZQtTSmk6DgAAwIhRJAGMsNs2D6TWZNWyBU1HAQAAGFGKJIARtm5Tf5YvmZULZ09vOgoAAMCIUiQBjKCHdz6Xh3c+n9XLDNkGAADGnrYWSaWU60opj5RStpRSPn6C53+9lPJgKeXeUsrXSikXtDMPQLut2zSQngkl1189v+koAAAAI65tRVIppSfJp5K8L8nSJDeVUpa+5LBNSVbUWq9O8oUk/7ldeQDabXCwZn1ff9526eycN2Ny03EAAABGXDuvSFqZZEut9fu11sNJ1iZZNfyAWus3aq0vDG1+J8miNuYBaKu7Hn8mO/YdzOrllrUBAABjUzuLpIVJtg3b3j6072R+MclfnuiJUspHSyl3l1Lu3r179whGBBg56/v6M31ST967dF7TUQAAANqiI4Ztl1J+LsmKJL99oudrrZ+uta6ota6YM2fO6IYDaMHBI8fy5ft25MevmJepk3qajgMAANAWvW187f4ki4dtLxra90NKKe9O8q+T/Fit9VAb8wC0zTcfeSrPHzxqWRsAADCmtfOKpI1JLi2lXFRKmZRkTZLbhh9QSlme5L8luaHW+lQbswC01a2b+jN7xuS86eLzmo4CAADQNm0rkmqtR5N8LMkdSR5Kckut9YFSyidLKTcMHfbbSWYk+Xwppa+UcttJXg6gY+174Ui+8fDu3HDNgvT2dMSKYQAAgLZo59K21FpvT3L7S/Z9Ytjjd7fz+wOMhtvv35HDxwazevmCpqMAAAC0lf86B3iF1m3qz6vmTM9VC89uOgoAAEBbKZIAXoH+vS/mrsefyeplC1NKaToOAABAWymSAF6B2/oGkiSrl7lbGwAAMPYpkgDOUK01t27antcumZUl501rOg4AAEDbKZIAztBDO57Po7v258blrkYCAADGB0USwBla39ef3gkl77/a3doAAIDxQZEEcAaODdas7xvIj716Ts6dPqnpOAAAAKNCkQRwBu56/OnsfO5gVlvWBgAAjCOKJIAzsG5Tf6ZP6sm7L5/bdBQAAIBRo0gCOE0HjxzLX963M9ddOT9TJ/U0HQcAAGDUKJIATtPXH34qzx86mtXLDdkGAADGF0USwGlat6k/c2ZOzpsunt10FAAAgFGlSAI4DXtfOJxvPPJUbrhmQXomlKbjAAAAjCpFEsBp+PJ9O3LkWM2N7tYGAACMQ4okgNOwftNALp4zPVcsOKvpKAAAAKNOkQTQou3PvpANTzyTG5cvTCmWtQEAAOOPIgmgRev7BpIkq5ZZ1gYAAIxPiiSAFtRas25Tf1ZccE4Wnzut6TgAAACNUCQBtODBHc/lsaf2Z7Uh2wAAwDimSAJowbpN/emdUPL+q+Y3HQUAAKAxiiSAUzg2WHPb5oG8/bLzc870SU3HAQAAaIwiCeAUvvP9p7PruUNZvXxB01EAAAAapUgCOIV1m/ozY3Jv3n353KajAAAANEqRBPAyDh45lr+8f2euu3JepkzsaToOAABAoxRJAC/jrx7alf2HjuZGd2sDAABQJAG8nHWbBjL3rMl5w6vOazoKAABA4xRJACfx7IHD+eYjT+WGaxakZ0JpOg4AAEDjFEkAJ/Hl+3bk6GDNasvaAAAAkiiSAE5q3ab+XHr+jCydf1bTUQAAADqCIgngBLY980LufvLZrF6+MKVY1gYAAJAokgBOaH1ff5Jk1bIFDScBAADoHIokgJeotebWTf1ZeeG5WXTOtKbjAAAAdAxFEsBLPDDwXL63+0BWLXc1EgAAwHCKJICXWLepPxN7St5/1fymowAAAHQURRLAMMcGa9ZvHsjbLzs/s6ZNajoOAABAR1EkAQzzd9/bk93PH8qNyxc2HQUAAKDjKJIAhlm3aSAzJ/fmna85v+koAAAAHUeRBDDkxcPH8pX7d+R9V83LlIk9TccBAADoOIokgCF/9dCuHDh8LKstawMAADghRRLAkHWb+jPvrCl5w0XnNR0FAACgIymSAJI8c+Bw/vrR3Vm1bEEmTChNxwEAAOhIiiSAJF++dyBHB2tWLbOsDQAA4GQUSQBJ1vUN5LK5M3P5/JlNRwEAAOhYiiRg3Nv69Av57pPPZtXyBSnFsjYAAICTUSQB4976vv4ksawNAADgFBRJwLhWa82tff15/UXnZuGsqU3HAQAA6GiKJGBcu69/X76/+0BWL3c1EgAAwKkokoBxbd2mgUzqmZCfuHJ+01EAAAA6niIJGLeOHhvMbZsH8o7XzMnZ0yY2HQcAAKDjKZKAcevvvvd09uw/lBstawMAAGiJIgkYt9Zt6s/MKb15+2XnNx0FAACgKyiSgHHphcNHc8cDO/P+q+ZnysSepuMAAAB0hbYWSaWU60opj5RStpRSPn6C599WSrmnlHK0lPIz7cwCMNydD+7KgcPHsmqZZW0AAACtaluRVErpSfKpJO9LsjTJTaWUpS85bGuSjyT5bLtyAJzI+r6BzD97Sl5/0blNRwEAAOga7bwiaWWSLbXW79daDydZm2TV8ANqrU/UWu9NMtjGHAA/5On9h/LXj+7ODcsWZMKE0nQcAACArtHOImlhkm3DtrcP7TttpZSPllLuLqXcvXv37hEJB4xfX7p3R44NVndrAwAAOE1dMWy71vrpWuuKWuuKOXPmNB0H6HLr+vrzmnkz85p5ZzUdBQAAoKu0s0jqT7J42PaioX0AjXliz4Fs2ro3q12NBAAAcNraWSRtTHJpKeWiUsqkJGuS3NbG7wdwSuv7BlJKcsM1C5qOAgAA0HXaViTVWo8m+ViSO5I8lOSWWusDpZRPllJuSJJSyrWllO1JPpDkv5VSHmhXHoBaa9b19ef1F52bBbOmNh0HAACg6/S288Vrrbcnuf0l+z4x7PHGHF/yBtB2927fl8f3HMgv/dirmo4CAADQlbpi2DbASLh1U38m9UzIdVfObzoKAABAV1IkAePC0WOD+dK9A3nX5efn7KkTm44DAADQlRRJwLjwrS17smf/4axa5m5tAAAAZ0qRBIwL6/sGctaU3rzjNXOajgIAANC1FEnAmHfg0NF85f6def/VCzK5t6fpOAAAAF1LkQSMeXc+uCsvHjmW1csWNB0FAACgqymSgDFvXV9/Fs6ammsvPLfpKAAAAF1NkQSMaXv2H8rfPrYnNyxbkAkTStNxAAAAupoiCRjTvrR5IMcGa25c7m5tAAAAr5QiCRjTbu0byOXzz8qr585sOgoAAEDXUyQBY9bjew5k87a9uXG5IdsAAAAjQZEEjFnrNvWnlOSGayxrAwAAGAmKJGBMqrVmfV9/3viq8zLv7ClNxwEAABgTFEnAmNS3bW+eePqFrF7maiQAAICRokgCxqT1fQOZ1Dsh1101r+koAAAAY4YiCRhzjhwbzBc3D+Q9l8/NWVMmNh0HAABgzFAkAWPOtx7bk6cPHM6qZe7WBgAAMJIUScCYs66vP7OmTczbLzu/6SgAAABjiiIJGFMOHDqarz6wKz9x1fxM6vUjDgAAYCR5lwWMKV99cGdePHIsNy53tzYAAICRpkgCxpRbNw1k4ayped2Sc5qOAgAAMOYokoAxY/fzh/Ktx3Zn9fIFmTChNB0HAABgzFEkAWPGFzcPZLAmq5dZ1gYAANAOiiRgzFjf158rFpyVS+fObDoKAADAmKRIAsaE7+3en83b9xmyDQAA0EaKJGBMWL+pP6UkP3nNgqajAAAAjFmKJKDr1Vqzrm8gb754duaeNaXpOAAAAGOWIgnoevds3Zutz7yQVctcjQQAANBOiiSg663v68/k3gm57sp5TUcBAAAY0xRJQFc7cmwwX7p3R969dG5mTpnYdBwAAIAxTZEEdLW/fWx3njlwODcuc7c2AACAdlMkAV3t1k0DmTVtYt726jlNRwEAABjzFElA19p/6GjufHBnrr96fib1+nEGAADQbt55AV3rjvt35uCRway2rA0AAGBUKJKAAK5/fwAACuxJREFUrrWurz+Lzpma111wTtNRAAAAxgVFEtCVnnruYL69ZU9uXL4wpZSm4wAAAIwLiiSgK922eSCDNVllWRsAAMCoUSQBXWl930CuWnh2Ljl/RtNRAAAAxg1FEtB1tjy1P/f178uqZQuajgIAADCuKJKArrO+rz8TSnLDNYokAACA0aRIArpKrTXr+vrz5ktm5/yzpjQdBwAAYFxRJAFd5Z6tz2bbMy9mtSHbAAAAo06RBHSVWzf1Z8rECfnxK+c1HQUAAGDcUSQBXePw0cF8+d4dec/SeZkxubfpOAAAAOOOIgnoGn/z6O48+8KR3LjckG0AAIAmKJKArrGurz/nTp+Ut146p+koAAAA45IiCegKzx88kjsf3JXrr56fiT1+dAEAADTBuzGgK3zl/p05dHQwq9ytDQAAoDGKJKArrO8byJJzp+W1S2Y1HQUAAGDcUiQBHW/Xcwfz7e/tyeplC1JKaToOAADAuKVIAjreFzcPpNZk1XLL2gAAAJqkSAI63q2b+nP1orNz8ZwZTUcBAAAY19paJJVSriulPFJK2VJK+fgJnp9cSvnc0PN3lVIubGceoPs8tuv5PDDwXFYbsg0AANC4thVJpZSeJJ9K8r4kS5PcVEpZ+pLDfjHJs7XWS5L8bpLfalceoDut6+vPhJJcf838pqMAAACMe71tfO2VSbbUWr+fJKWUtUlWJXlw2DGrkvy7ocdfSPJ7pZRSa61tzNURfv2Wvhw6Oth0DOh4f/+9p/OWS+fk/JlTmo4CAAAw7rWzSFqYZNuw7e1JXn+yY2qtR0sp+5Kcl2TP8INKKR9N8tEkWbJkSbvyjqpHdz2fFw8fazoGdLzZMyblH7/1oqZjAAAAkPYWSSOm1vrpJJ9OkhUrVoyJq5W+9CtvbToCAAAAwGlp57Dt/iSLh20vGtp3wmNKKb1Jzk7ydBszAQAAAHCG2lkkbUxyaSnlolLKpCRrktz2kmNuS/Lhocc/k+Tr42E+EgAAAEA3atvStqGZRx9LckeSniR/VGt9oJTyySR311pvS/KHSf6/UsqWJM/keNkEAAAAQAdq64ykWuvtSW5/yb5PDHt8MMkH2pkBAAAAgJHRzqVtAAAAAIwhiiQAAAAAWqJIAgAAAKAliiQAAAAAWqJIAgAAAKAliiQAAAAAWqJIAgAAAKAliiQAAAAAWqJIAgAAAKAliiQAAAAAWqJIAgAAAKAliiQAAAAAWqJIAgAAAKAliiQAAAAAWqJIAgAAAKAliiQAAAAAWqJIAgAAAKAliiQAAAAAWqJIAgAAAKAlpdbadIbTUkrZneTJpnOMkNlJ9jQdgo7k3OBknBu8HOcHJ+Pc4GScG5yMc4OX4/wYmy6otc451UFdVySNJaWUu2utK5rOQedxbnAyzg1ejvODk3FucDLODU7GucHLcX6Mb5a2AQAAANASRRIAAAAALVEkNevTTQegYzk3OBnnBi/H+cHJODc4GecGJ+Pc4OU4P8YxM5IAAAAAaIkrkgAAAABoiSKpAaWU60opj5RStpRSPt50HjpHKWVxKeUbpZQHSykPlFJ+relMdJZSSk8pZVMp5UtNZ6FzlFJmlVK+UEp5uJTyUCnljU1nojOUUv7Z0O+T+0spN5dSpjSdieaUUv6olPJUKeX+YfvOLaXcWUp5bOjzOU1mpBknOTd+e+j3yr2llFtLKbOazEhzTnR+DHvuN0optZQyu4lsNEORNMpKKT1JPpXkfUmWJrmplLK02VR0kKNJfqPWujTJG5L8svODl/i1JA81HYKO8/8k+Uqt9TVJrolzhCSllIVJfjXJilrrlUl6kqxpNhUN+5Mk171k38eTfK3WemmSrw1tM/78SX703LgzyZW11quTPJrkX412KDrGn+RHz4+UUhYneW+SraMdiGYpkkbfyiRbaq3fr7UeTrI2yaqGM9Ehaq07aq33DD1+PsffDC5sNhWdopSyKMn7k/xB01noHKWUs5O8LckfJkmt9XCtdW+zqeggvUmmllJ6k0xLMtBwHhpUa/2bJM+8ZPeqJH869PhPk6we1VB0hBOdG7XWr9Zajw5tfifJolEPRkc4yc+OJPndJP8yicHL44wiafQtTLJt2Pb2KAo4gVLKhUmWJ7mr2SR0kP87x39ZDzYdhI5yUZLdSf54aNnjH5RSpjcdiubVWvuT/Jcc/5/iHUn21Vq/2mwqOtDcWuuOocc7k8xtMgwd6xeS/GXTIegcpZRVSfprrZubzsLoUyRBByqlzEjy50n+aa31uabz0LxSyvVJnqq1frfpLHSc3iSvTfL7tdblSQ7E0hSSDM26WZXjZeOCJNNLKT/XbCo6WT1+O2dXFvBDSin/OsfHL3ym6Sx0hlLKtCT/e5JPNJ2FZiiSRl9/ksXDthcN7YMkSSllYo6XSJ+ptf5F03noGG9OckMp5YkcXxL7zlLKnzUbiQ6xPcn2WusPrl78Qo4XS/DuJI/XWnfXWo8k+Yskb2o4E51nVyllfpIMfX6q4Tx0kFLKR5Jcn+Rnh4pGSJKLc/w/KTYP/W26KMk9pZR5jaZi1CiSRt/GJJeWUi4qpUzK8aGXtzWciQ5RSik5PufkoVrr7zSdh85Ra/1XtdZFtdYLc/znxtdrra4sILXWnUm2lVIuG9r1riQPNhiJzrE1yRtKKdOGfr+8Kwax86NuS/LhoccfTrK+wSx0kFLKdTm+pP6GWusLTeehc9Ra76u1nl9rvXDob9PtSV479DcJ44AiaZQNDaz7WJI7cvyPuVtqrQ80m4oO8uYk/1uOX23SN/TxE02HAjreryT5TCnl3iTLkvzHhvPQAYauUvtCknuS3Jfjf/d9utFQNKqUcnOSv09yWSlleynlF5P8pyTvKaU8luNXsf2nJjPSjJOcG7+XZGaSO4f+Jv1/Gw1JY05yfjCOFVcoAgAAANAKVyQBAAAA0BJFEgAAAAAtUSQBAAAA0BJFEgAAAAAtUSQBAAAA0BJFEgBAi0op/7qU8kAp5d6h22G/vpTyT0sp05rOBgAwGkqttekMAAAdr5TyxiS/k+TttdZDpZTZSSYl+bskK2qtexoNCAAwClyRBADQmvlJ9tRaDyXJUHH0M0kWJPlGKeUbSVJKeW8p5e9LKfeUUj5fSpkxtP+JUspvlVI2DH1cMrT/A6WU+0spm0spf9PMPw0AoDWuSAIAaMFQIfStJNOS/FWSz9Va/7qU8kSGrkgaukrpL5K8r9Z6oJTym0km11o/OXTcf6+1/odSys8n+WCt9fpSyn1Jrqu19pdSZtVa9zbyDwQAaIErkgAAWlBr3Z/kdUk+mmR3ks+VUj7yksPekGRpkm+XUvqSfDjJBcOev3nY5zcOPf52kj8ppfzjJD3tSQ8AMDJ6mw4AANAtaq3HknwzyTeHriT68EsOKUnurLXedLKXeOnjWusvlVJen+T9SfpKKctqrU+PbHIAgJHhiiQAgBaUUi4rpVw6bNeyJE8meT7JzKF930ny5mHzj6aXUl497Gs+NOzz3w8dc3Gt9a5a6yeS7EmyuI3/DACAV8QVSQAArZmR5L+WUmYlOZpkS44vc7spyVdKKQO11ncMLXe7uZQyeejr/k2SR4ceTy6l3JXj/5n3g6uWfnuooCpJvpZk86j8awAAzoBh2wAAo2D4UO6mswAAnClL2wAAAABoiSuSAAAAAGiJK5IAAAAAaIkiCQAAAICWKJIAAAAAaIkiCQAAAICWKJIAAAAAaIkiCQAAAICW/P8JFu4/OReaSgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x29c03067438>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create the environment\n",
    "#env = gym.make('Taxi-v2')\n",
    "env = gym.make(\"FrozenLake-v0\")\n",
    "obs = env.reset()\n",
    "\n",
    "obs_length = env.observation_space.n\n",
    "n_actions = env.action_space.n\n",
    "\n",
    "reward_count = 0\n",
    "games_count = 0\n",
    "\n",
    "# Create and initialize the table with 0.0\n",
    "table = collections.defaultdict(float)\n",
    "    \n",
    "test_rewards_list = []\n",
    "\n",
    "while games_count < MAX_GAMES:\n",
    "\n",
    "    # Select the action following an ε-greedy policy\n",
    "    action = select_eps_greedy_action(table, obs, n_actions)\n",
    "    next_obs, reward, done, _ = env.step(action)\n",
    "\n",
    "    # Update the Q-table\n",
    "    Q_learning(table, obs, next_obs, reward, action)\n",
    "\n",
    "    reward_count += reward\n",
    "    obs = next_obs\n",
    "\n",
    "    if done:\n",
    "        epsilon *= EPS_DECAY_RATE\n",
    "\n",
    "        # Test the new table every 1k games\n",
    "        if games_count % 1000 == 0:\n",
    "            test_reward = test_game(env, table, n_actions)\n",
    "            print('\\tEp:', games_count, 'Test reward:', test_reward, np.round(epsilon,2))\n",
    "\n",
    "            test_rewards_list.append(test_reward)\n",
    "\n",
    "        obs = env.reset()\n",
    "        reward_count = 0\n",
    "        games_count += 1    \n",
    "\n",
    "# Plot the accuracy over the number of steps\n",
    "plt.figure(figsize=(20,10))\n",
    "plt.xlabel('Steps')\n",
    "plt.ylabel('Accurracy')\n",
    "plt.plot(test_rewards_list)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 注意: 如果你要Q-learning有一个连续的策略和动作，可以适当调整策略和动作的步长"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
